4. Experimentation: the Effect of Exploration on the Agent's Performance

Model-based learning of interaction strategies in repeated-games has received a lot of attention in the game-theory literature. Gilboa & Samet [10] deal with bounded regular players. They describe a model-based learning strategy for repeated games that learns the best response against any regular strategy. Their procedure enumerates the set of all automata and chooses the current opponent model to be the first automaton in the sequence that is consistent with the current history. Exploration is achieved by designing a sequence of actions that distinguishes between the current model and the next consistent automaton in the enumera-tion. The risk involved in exploration is bypassed by assuming that the opponents' strategies are limited to strongly connected automata, where there are no " sinks " and there is always opportunities to regret. For such automata, the learning algorithm is guaranteed to converge to the best response in the limit. This learning procedure is based on exhaustive search in the space of automata, and therefore, is impractical for computational bounded agents. The main role of the opponent model is to predict its behavior in the future. Choosing a proper class of strategies for modeling is essential for the success of the model-based strategy. If the model class is too restricted it will probably fail in prediction. On the other hand, a too general class will make the best response problem and the learning problem intractable. Often, there are many ways to model a given behavior. This paper concentrates on deterministic finite automata for modeling the agents' strategies. The question how the model-based framework can be extended for more powerful agents remains open for future research. The complexity of computing a best response automaton in repeated games with mixed strategies. Efficient algorithms for learning to play repeated games against computationally bounded adversaries. actions to the given history, and by applying the learning algorithm to the expanded histories, we acquire models that are consistent with the history and predict differently the opponent responses for the player sequences of actions. To summarize, for exploring the opponent's strategy using a mixed model, the agent first searches d stages forward for collecting different opponent models to the set of support. It then infers a belief distribution over this set. Following that, it finds the-best response against the mixed model and performs a sequence of actions dictated by this strategy. By doing so, the agent rationally balances between exploration and …